AFCAPS-TR-201 6-0001  technical  report  on  cross- 

validation  OF  AFOQT  FORM  S  FOR 
CYBERSPACE  OPERATIONS  - 
CYBERSPACE  CONTROL 


April  20,  2016 


Gregory  G  Manley,  Ph.D. 

Air  Force  Personnel  Center 
Strategic  Research  and  Assessment  Branch 
Randolph  AFBTX  78150 


Kenneth  L.  Schwartz 
Johnny  J.  Weissmuller 

AFPC/Strategic  Research  and 
Assessment  Branch  (DSYX) 


Air  Force  Personnel  Center 
Strategic  Research  and  Assessment 
HQ  AFPC/DSYX 
550  C  Street  West,  Ste  45 
Randolph  AFB  TX  78150-4747 


Approved  for  Public  Release.  Distribution  Unlimited 

UNCLASSIFIED 


Approved  for  Public  Release.  Distribution  Unlimited 

UNCLASSIFIED 


NOTICE 


When  Government  drawings,  specifications,  or  other  data  are  used  for  any  purpose  other 
than  in  connection  with  a  definitely  Government-related  procurement,  the  United  States 
Government  incurs  no  responsibility  or  any  obligation  whatsoever.  The  fact  that  the 
Government  may  have  formulated  or  in  any  way  supplied  the  said  drawings,  specifications, 
or  other  data,  is  not  to  be  regarded  by  implication,  or  otherwise  in  any  manner  construed,  as 
licensing  the  holder,  or  any  other  person  or  corporation;  or  as  conveying  any  rights  or 
permission  to  manufacture,  use,  or  sell  any  patented  invention  that  may  in  any  way  be 
related  thereto. 


This  report  was  cleared  for  release  by  HQ  AFPC/DSYX  Strategic  Research  and  Assessment 
Branch  and  is  releasable  to  the  Defense  Technical  Information  Center.  Available  for  public 
release.  Distribution  Unlimited.  Please  contact  AFPC/DSYX  Strategic  Research  and 
Assessment  with  any  questions  or  concerns  with  the  report. 


This  report  is  published  as  received  with  minor  grammatical  corrections.  The  views 
expressed  are  those  of  the  authors  and  not  necessarily  those  of  the  United  States 
Government,  the  United  States  Department  of  Defense,  or  the  United  States  Air  Force.  In  the 
interest  of  expediting  publication  of  impartial  statistical  analysis  of  Air  Force  tests  SRAB  does 
not  edit  nor  revise  Contractor  assessments  appropriate  to  the  private  sector  which  do  not 
apply  within  military  context. 


Federal  Government  agencies  and  their  contractors  registered  with  Defense  Technical 
Information  Center  should  direct  request  for  copies  of  this  report  to: 

Defense  Technical  Information  Center  -  http://www.dtic.mil/ 


This  paper  has  been  reviewed  by  the  Air  Force  Center  for  Applied  Personnel  Studies 
(AFCAPS)  and  is  approved  for  publication.  AFCAPS  members  include:  Senior  Editor  Dr. 
Thomas  Carretta  AFMC  71 1  HPW/RHCI,  Associate  Editor  Dr  Gregory  G.  Manley,  AFPC/DSYX. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including 
suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway, 
Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of 
information  if  it  does  not  display  a  currently  valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1 .  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE  3.  DATES  COVERED  (From  -  To) 

04-20-2016  Technical  2012-2014 


4.  TITLE  AND  SUBTITLE 

TECHNICAL  REPORT  ON  CROSS-VALIDATION  OF  AFOQT  FORM  S 
FOR  CYBERSPACE  OPERATIONS  -  CYBERSPACE  CONTROL 


6.  AUTHOR(S) 

Gregory  G.  Manley,  Ph.D. 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Personnel  Center,  HQ  AFPC/DSYX 
550  C  Street  W 
Randolph  AFB,  TX  78150 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Personnel  Center 

Strategic  Research  and  Assessment  Branch 

Randolph  AFB  TX  78150 


12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Further  distribution  only  as  directed  by  AFPC/DSYX 
Assessment  Branch  or  higher  DoD  authority 


13.  SUPPLEMENTARY  NOTES 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 

HQ  AFPC/DSYX 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

AFCAPS-FR-2 01 6-0001 


Strategic  Research  and 


14.  ABSTRACT 

This  study  examined  the  relationship  and  predictive  value  of  scores  from  the  AFOQT  Form  S  with  Final  School  Grade  (FSG)  for  Air 
Force  Officer  Cyberspace  Operations  /  Cyberspace  Control  School  (17D  Undergraduate  Cyber  Training  Phase  II).  Form  S  scores 
include  officer  aptitude  composites  (e.g.,  Pilot  Composite),  individual  sub-tests  (e.g.,  Word  Knowledge),  and  Domain  (e.g., 
Conscientiousness)  and  Facet  [e.g.,  Achievement-Striving)  personality  scales  from  the  Self-Description  Inventory  Plus  (SDI+).  Data 
from  all  non-prior  service  U.S.  Air  Force  Officers  who  attended  the  school  from  November  2010  to  March  2012  (N=295)  were 
analyzed  and  results  are  reported  herein.  The  sample  was  separated  by  quintiles  on  FSG  and  results  are  displayed  for  each 
predictor  by  quintile.  The  number  of  attrits  was  calculated  for  each  quintile,  showing  a  total  of  only  12,  of  which  10  fell  in  the 
bottom  two  quintiles  and  none  in  the  top  quintile.  All  composites  and  sub-tests  were  significantly  correlated  to  FSG  as  were  many 
of  the  domain  and  facet  scales  of  the  SDI+.  A  model  of  the  best  combination  of  individual  predictors  was  determined  through  a 
series  of  multiple  regression  analyses.  Recommendations  for  a  best-use  selection  model  and  cut-scores  are  made. 


15.  SUBJECT  TERMS 

CYBERSPACE  OPERATIONS  -  CYBERSPACE  CONTROL,  17D,  Regression  modeling.  Cross-validation, 
USAF 


16.  SECURITY  CLASSIFICATION  OF: 

Unclassified 


a.  REPORT 

U 


b.  ABSTRACT 

U 


c.  THIS  PAGE 

U 


17.  LIMITATION 
OF  ABSTRACT 

18. 

NUMBER 

OF  PAGES 

U 

30 

19a.  NAME  OF  RESPONSIBLE  PERSON 

Kenneth  L.  Schwartz 


19b.  TELEPHONE  NUMBER  (include  area  code) 

210-565-3139 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39.1 8 


TECHNICAL  REPORT  ON  CROSS-VALIDATION  OF  AFOQT  FORM  S  FOR 
CYBERSPACE  OPERATIONS  -  CYBERSPACE  CONTROL 

(AFSC-AWARDING  17D  PHASE  II  FINAL  SCHOOL  GRADE) 


GREGORY  G.  MANLEY,  PHD 
AF/A1PT 
AFPC/DSYX 
JUNE  2014 


SUMMARY 


This  study  examined  the  relationship  and  predictive  value  of  scores  from  the  Air  Force  Officer  Qualifying 
Test  (AFOQT)  Form  S  with  Final  School  Grade  (FSG)  for  Air  Force  Officer  Cyberspace  Operations  / 
Cyberspace  Control  School  (17D  Undergraduate  Cyber  Training  Phase  II).  Form  S  scores  include  officer 
aptitude  composites  ( e.g .,  Pilot  Composite),  individual  sub-tests  ( e.g .,  Word  Knowledge),  and  Domain 
(i e.g .,  Conscientiousness)  and  Facet  {e.g.,  Achievement-Striving)  personality  scales  from  the  Self 
Description  Inventory  Plus  (SDI+).  Data  from  all  non-prior  service  U.S.  Air  Force  Officers  who  attended 
the  school  from  November  2010  to  March  2012  (N=295)  were  analyzed  and  results  are  reported  herein. 
The  sample  was  separated  by  quintiles  on  FSG  and  results  are  displayed  for  each  predictor  by  quintile. 
The  number  of  attrits  was  calculated  for  each  quintile,  showing  a  total  of  only  12,  of  which  10  fell  in  the 
bottom  two  quintiles  and  none  in  the  top  quintile.  All  composites  and  sub-tests  were  significantly 
correlated  to  FSG  as  were  many  of  the  domain  and  facet  scales  of  the  SDI+.  A  model  of  the  best 
combination  of  individual  predictors  was  determined  through  a  series  of  multiple  regression  analyses. 
This  model  was  cross-validate  with  subsequent  samples  and  revised.  Recommendations  for  a  best-use 
selection  model  and  cut-scores  are  made. 
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PREDICTIVE  VALIDATION  OF  AFOQT  FORM  S  FOR  CYBERSPACE  OPERATIONS  -  CYBERSPACE  CONTROL 
(AFSC-AWARDING  17D  PHASE  II  FINAL  SCHOOL  GRADE) 


The  17D  Air  Force  Specialty  (AFS)  Phase  II  School  is  the  AFS-awarding  school  for  officer  Undergraduate 
Cyber  Training  (UCT).  In  this  AFS,  officers  are  trained  to  execute  the  functions  and  activities  of 
cyberspace  and  information  operations.  In  the  field  they  plan,  organize,  and  direct  operations;  including 
network  attack,  network  defense,  network  warfare  support,  network  operations,  and  related 
information  operations.  Such  operations  cover  the  spectrum  of  mission  areas  within  the  domain  of 
cyberspace.  17D  UCT  encompasses  two  primary  curricula,  Cyber  Defense  and  Cyber  Control.  The  former 
includes  activities  such  as  planning,  organizing,  and  performing  network  defense;  and  exploitation  and 
attack  in  support  of  mission  objectives.  The  latter  involves  the  planning,  organization,  and  performance 
of  network  operations  to  include  establishment,  operations,  and  information  assurance  and  defense  in 
support  of  mission  objectives.  At  the  time  these  Final  School  Grades  (FSGs)  were  observed,  the  training 
encompassed  both  curricula,  however,  specialty  shred-outs  have  since  been  designated  for  the  Cyber 
Defense  (17DXA)  and  Cyber  Control  (17DXB)  Air  Force  Specialty  Codes  (AFSCs). 

Currently,  there  is  no  formal  test  for  selecting  officers  for  training  in  17D  UCT  nor  are  there  any  cut- 
scores  in  use  on  any  AFOQT  composite  other  than  those  for  becoming  an  Air  Force  officer.  Selection  into 
the  17D  AFS  is  conducted  through  a  computer  program  that  examines  the  candidates'  required 
undergraduate  coursework.  A  Bachelor  of  Science  in  an  appropriate  undergraduate  major  for  17D  UCT 
are  considered  ( e.g .,  Computer  Science,  Information  Technology)  as  are  other  Bachelor  of  Science 
majors  if  the  candidate  has  accomplished  at  least  24  credit  hours  of  appropriate  sciences  courses  {e.g., 
Telecommunications,  Engineering,  Information  Assurance).  This  study  was  conducted  to  determine  if 
any  test  scores  available  from  Form  S  of  the  AFOQT  were  predictive  of  17D  UCT  performance  and  course 
completion;  and  to  develop  a  predictor  model  with  cut-score  recommendations. 


METHOD 


Initial  Validation  Sample 

Data  included  295  usable  cases  of  all  non-prior  service  (NPS;  directly  accessing  from  college)  U.S.  Air 
Force  officers  attending  the  school  between  the  dates  of  November,  2010  (inception  of  UCT)  and  March, 
2012.  Any  senior  U.S.  Air  Force  officers  and  other  non-U. S.  Air  Force  officer  cases  were  removed 
including  foreign  nationals,  civilians,  enlisted  Air  Force  and  other  U.S.  Services  personnel.  Of  the  295  NPS 
U.S.  Air  Force  officer  cases,  Phase  II  attrition  was  low,  around  4%  as  only  12  did  not  fully  complete  the 
course.  Most  attrition  appears  to  be  related  (in  part  at  least)  to  academic  performance,  with  8  falling  in 
the  bottom  quintile  based  on  FSG  and  0  in  the  top.  Note  that  17D  UCT  Phase  I  is  a  qualifying  course  for 
Phase  II  where  some  initial  attrition  occurred  (5  of  300  NPS  Air  Force  officer  students  or  1.7%).  Overall 
combined  attrition  for  both  Phase  I  and  II  was  5.7%. 

Re-  and  Cross-validation  samples 

An  additional  291 17D  students  were  added  to  the  sample  who  attended  the  training  from  April  2012  to 
January  2014.  The  subsequent  sample  (N=291)  was  used  for  initial  cross-validation.  For  further  re¬ 
validation  and  cross-validation  purposes,  the  full  sample  (N=586)  was  randomly  assigned  to  two  groups 
(re-validation  and  cross-validation  samples)  using  a  bi-fold  design. 
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The  total  sample  was  mostly  male  (81%),  ethnically  non-Hispanic  (89.5%)  and  White  (72%).  Other  races 
reported  included  Asian  (14%),  Black  or  African-American  (9.8%),  Native  American  or  Alaskan  (2.7%), 
and  Pacific  Islander  (1.4%). 

Measures  of  the  AFOQT-  Form  5 

Side  1  of  the  AFOQT  Form  S  contains  various  subtest  measures  of  general  and  specific  cognitive  abilities 
from  which  composite  scores  are  computed.  These  include  Verbal  Analogies  (VA),  Arithmetic  Reasoning 
(AR),  Word  Knowledge  (WK),  Math  Knowledge  (MK),  Instrument  Comprehension  (1C),  Block  Counting 
(BC),  Table  Reading  (TR),  Aviation  Information  (Al),  General  Science  (GS),  Rotated  Blocks  (RB),  and 
Flidden  Figures  (HF).  The  composites  are  used  for  selection  into  rated  officership  (i.e.,  pilots)  and  as 
minimums  for  becoming  an  Air  Force  officer.  The  composites  are  labeled  PILOT,  NAVIGATOR- 
TECHNICIAN,  ACADEMIC,  VERBAL,  and  QUANTITATIVE.  Thompson,  Skinner,  Gould,  Alley,  &  Shore 
(2010)  provided  a  full  description  of  the  subtest  and  the  computed  composites.  Reliabilities  for  the 
subtests  are  typically  acceptable  to  good  in  the  normative  sample  (as  ranging  between  .71  and  .90)  and 
are  reported  in  Morath,  Parish,  and  Lodato  (2010).  Parish,  Morath,  Lodato,  and  Stachowski  (2010) 
conducted  test-retest  reliability  analyses  for  the  composites  from  a  sample  of  Air  Force  applicants  who 
took  the  same  forms  twice  and  found  them  to  be  moderately  high,  with  coefficients  ranging  between 
.82  and  .89. 

Side  2  of  Form  S  contains  self-report  personality  trait  items  of  the  Self  Description  Inventory  Plus  (SDI+), 
an  inventory  designed  to  measure  the  factors  of  the  Big  Five  as  well  as  Teamwork  Orientation  and 
Service-Before-Self  Orientation  (T upes  &  Christal,  1961;  Thompson  et  al.,  2010).  Manley  (2011) 
conducted  a  factor  analyses  based  on  roughly  60,000  U.S.  Air  Force  applicants  to  develop  Domain- 
(broad)  and  Facet-  (narrow)  level  scales  for  the  SDI+.  The  Domain  scales  include  the  Big  Five 
(Agreeableness,  Neuroticism,  Introversion-Extraversion,  Conscientiousness,  &  Openness )  and  a  smaller 
sixth  factor  called  Machiavellianism  (Mach).  The  Facet  scales  were  also  factorially  derived  by  separately 
factor  analyzing  the  Domain  factors  producing  20  facets  in  total.  These  Domains  and  Facets  are  listed  in 
Tables  4-9.  Reliabilities  for  theses  scales  are  generally  acceptable  to  very  good  for  the  normative 
sample,  ranging  in  the  .70s  to  high  .90s  for  most  scales,  and  are  reported  in  Manley  (2011). 

Procedure 

The  AFOQT  Form  S  predictor  data  and  the  17D  UCT  school  FSG  data  were  submitted  to  descriptive 
statistical  and  correlational  analysis  to  compute  means  (Ms),  standard  deviations  (SDs),  and  a 
correlation  matrix  for  the  study  variables.  A  distribution  histogram  of  FSG  (overall  percent  correct)  was 
plotted  and  is  shown  in  Figure  1.  The  FSG  data  appears  mostly  normally  distributed  with  a  negative  skew 
to  the  lower  end  of  the  distribution  due  to  range  restriction  as  most  scores  are  distributed  about  the 
upper  range  of  possible  scores.  Quintiles  on  the  FSG  variable  were  determined  so  that  the  n-size  for 
each  group  was  roughly  equal  (the  quintile  cut  points  used  produced  the  most  evenly  numbered  groups 
possible).  Figure  1  graphically  displays  the  quintile  breakdown  among  the  distribution  of  FSG  scores. 
Table  1  reports  the  n-sizes,  range  of  scores,  and  descriptive  statistics  (Ms,  SDs)  for  the  quintiles. 

Using  multiple  regression  analysis,  FSG  scores  were  regressed  on  different  combinations  of  significant 
predictors  (zero-order  correlations  p  <  .05)  to  determine  the  best  model  in  terms  of  variance  predicted, 
parsimony,  and  unique  contribution  of  each  variable  to  predicting  FSG  variance. 
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Possible  cut-scores  were  examined  for  optimal  prediction  with  minimal  decision  errors  (Type  I  or  Type  II) 
by  applying  the  recommended  weights  from  the  optimal  predictor  model  to  scores  from  the  sample.  Chi 
Square  analysis  was  used  to  determine  hit  and  miss  rates  of  the  model  predicting  FSG  above  or  below  a 
FSG  score  of  80  (FSG=80  was  used  as  a  conservative  base  rate  of  "success"  because  the  model  was 
developed  on  the  same  sample  and  most  students  in  the  sample  passed  the  course)  and  Phi  coefficients 
(4>)  were  examined  to  determine  the  best  fitting  model  cut-score.  Standardized  mean  score  differences 
(Cohen's  d )  for  EEOC  protected  subgroups  were  examined. 


RESULTS 

The  Ms,  SDs,  and  criterion-related  validities  (rxy)  for  each  predictor  that  were  observed  from  the 
correlation  matrix  are  reported  in  Tables  2-9.  The  Ms  for  each  predictor  by  FSG  quintile  are  also 
reported  in  Tables  2-9  and  graphically  displayed  in  Figures  2-9. 

Composites 

All  AFOQT  composites  had  significant  uncorrected  criterion-related  validities  with  FSG  (p<.001).  The 
uncorrected  validities  (rs)  ranged  from  .29  (QUANTITATIVE)  to  .38  (ACADEMIC)  and  averaged  .35.  Table 
2  reports  these  validities  and  the  mean  composite  percentile  scores  for  each  quintile.  Both  Table  2  and 
Figure  2  display  the  mean  composite  scores  by  quintile  as  monotonically  increasing.  The  average  score 
for  the  first  (bottom)  to  the  fifth  (top)  quintiles  are  respectively,  50,  61,  64,  67,  and  80.  These  validity 
and  mean  quintile  results  indicate  all  composites  are  highly  valid,  positive  linear  predictors  of  overall 
17D  UCT  academic  performance. 

Sub-Tests 

As  was  the  case  with  the  composites,  all  AFOQT  sub-tests  had  significant  uncorrected  criterion-related 
validities  with  FSG  (p  <  .001  or  p  <  .01).  The  validities  ranged  from  .19  (BC,  TR)  to  .34  (WK,  GS)  and 
averaged  .26.  Table  3  reports  these  results  and  the  mean  scores  by  quintiles  are  displayed  in  Table  3  and 
Figure  3.  Again,  the  quintile  scores  are  monotonically  increasing  for  all  predictors,  which  along  with  the 
significant  validities,  indicate  the  sub-tests  are  valid  (highly  valid  in  some  cases),  positive  linear 
predictors  of  overall  17D  UCT  academic  performance. 

SDI  Domains 

Table  4  reports  the  uncorrected  criterion-related  validities  for  the  Domain  personality  scales  with  FSG. 
Non-cognitive  scales  are  not  expected  to  be  as  highly  predictive  of  performance  in  a  cognitive  domain 
(academic  performance)  but  are  hoped  to  predict  some  unique  variance  of  school  performance  (or 
attrition)  due  to  non-cognitive  personal  characteristics  (e.g.,  stress  tolerance,  teamwork  orientation,  or 
motivation).  Here,  three  of  the  six  scales  were  significantly  correlated  with  FSG.  Neuroticism  (r  =  -.21,  p  < 
.001)  and  Conscientiousness  (r  =  .19,  p  <  .001)  were  the  two  best  Domain  predictors  with  Agreeableness 
also  significantly  predicting  FSG  (r  =  .12,  p  <  .05).  From  an  examination  of  Table  4  and  Figure  4,  the 
Domain  scale  means  by  quintile  are  monotonic  (increasing  or  decreasing  in  the  expected  direction)  for 
the  significant  predictor  domains,  indicating  a  valid  linear  relationship  with  FSG.  However,  two  of  the 
non-significant  domains  appear  to  show  a  nonlinear  relationship  with  FSG,  which  would  result  in  poor 
validity  when  scaled  as  linear.  Introversion-Extraversion  (high  scores  indicate  Introversion)  shows  lower 
scores  for  bottom  and  top  FSG  quintiles  and  Openness  shows  higher  scores  for  bottom  and  top  quintiles. 
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The  graphs  suggest  that  these  two  scales  may  be  predictive  if  modeled  in  an  appropriate  nonlinear  way. 
Individual  follow-up  regression  analyses  modeling  these  variables  as  quadratic  (as  shown  below), 
however,  showed  no  significant  prediction. 


FSG  =  b0  +  biX  +  b2X2 ...  +  £ 


Where  X  is  the  predictor  test  and  bo,  b  1,  and  b2  are  the  weights  for  the  intercept,  linear,  and  quadratic 
terms,  respectively.  A  cubed  term  (b3X3)  is  added  to  the  equation  for  a  cubic  polynomial  regression.  The 
£  term  represents  normally  distributed  random  error. 

SDI  Facets 

The  SDI  domains  each  have  a  number  of  sub-factors  called  facets  associated  with  them.  The  number  of 
facets  per  domain  vary,  ranging  from  two  (Conscientiousness)  to  five  (Agreeableness),  and  total  20 
facets  for  all  six  domains.  Eight  of  these  facets  had  significant  uncorrected  criterion-related  validities 
with  FSG. 

The  Agreeableness  domain  has  five  facets,  two  of  which  had  significant  validities  with  FSG,  Team  Player 
(Al;  r=.19,  p<.001)  and  Hyper-Competitive  (A5;  r=-.12,  p<. 05).  Table  5  reports  the  results  for  the 
Agreeableness  facets.  As  shown,  the  facets  of  Pleasant  (A2),  Considerate  (A3),  and  Helpful-Altruistic  (A4) 
did  not  significantly  correlate  to  FSG,  however,  Pleasant  was  nearly  as  predictive  (r=.ll)  as  Hyper- 
Competitive.  Figure  5  displays  graphically  the  relationship  between  the  facets  and  FSG  quintile.  As 
shown,  the  facet  of  Hyper-Competitive  appears  to  have  somewhat  of  a  non-linear  relationship  with  FSG 
in  that  the  relatively  higher  scores  tend  to  be  grouped  in  the  middle  quintiles  (especially  the  second 
quintile)  while  the  lowest  scores  tend  to  appear  in  the  highest  (fifth)  quintile.  Predictability  of  FSG  for 
this  facet  may  be  improved  by  appropriately  modeling  both  linear  and  non-linear  relationships  with  the 
criterion.  As  was  the  case  with  the  Introversion-Extraversion  and  Openness  domain  scales  though, 
quadratic  (or  polynomial)  regressions  were  non-significant. 

Neuroticism  has  three  facets,  Stress-Under-Pressure  (Nl;  r=-.22,  p<.001),  Temperamental  (N2;  r=-.17, 
p<.01),  and  Worry  (N3;  r=-.13,  p<. 05),  and  all  had  significant  validities  with  FSG.  Table  6  and  Figure  6 
display  these  results.  The  quintiles  bar  graph  indicates  a  generally  linear  relationship  of  these  facets  with 
FSG.  Note  the  Stress-Under-Pressure  facet  has  essentially  the  same  validity  as  the  Neuroticism  domain 
scale  (r=-.21),  indicating  that  this  facet  is  the  primary  driver  of  the  domain  scale  in  predicting  FSG. 

Table  7  reports  the  validities  for  the  three  Introversion-Extraversion  facets.  Note  that  while  high  scores 
on  the  IE  domain  scale  indicate  Introversion,  high  scores  on  Dominance  and  Sociability  facet  scales 
indicate  Extraversion.  Of  the  IE  facet  scales,  only  the  Dominance  (IE3)  facet  was  significantly  related  to 
FSG  (r=.13,  p<. 05).  The  facets  of  Unassertive  (IE1)  and  Sociable  (IE2)  did  not  significantly  predict  the 
criterion;  however,  an  examination  of  Figure  7  shows  the  Sociable  facet  to  be  somewhat  nonlinear  in  its 
relationship  with  FSG  in  that  those  scoring  relatively  higher  on  this  scale  are  either  in  the  top  or  bottom 
quintiles  while  those  scoring  relatively  lower  reside  in  the  middle  quintiles.  This  graph  suggests  it  is 
possible  that  this  measure  is  significantly  predictive  of  FSG  when  scaled  in  an  appropriate  nonlinear  way. 
Again,  FSG  was  regressed  on  the  linear  and  quadratic  terms  for  the  Sociable  scale  and  resulted  in  a 
significant  prediction  model  (/?=.  19)  with  all  terms  statistically  significant  (intercept,  linear,  and  quadratic 
terms;  p  <  .005).  This  result  is  interpreted  as  those  lower  in  Sociability  generally  tend  to  do  better  than 
mid-level  Sociability,  but  those  very  high  in  Sociability  also  do  well.  Table  7  shows  this  scale  was  a  non- 
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significant  linear  predictor  of  FSG  with  an  observe  effect  size  of  r  =  -.03,  hence  non-linear  scaling  of  this 
variable  greatly  improves  its  predictability. 

Results  for  the  two  Conscientiousness  facets  are  reported  in  Table  8.  Achievement-Striving  (Cl) 
significantly  predicted  FSG  (r=.21,  p<.001)  and  while  Order  (C2)  was  non-significant,  a  small  effect  was 
observed  nonetheless  (r=.10).  Figure  8  clearly  indicates  a  very  linear  pattern  for  Achievement-Striving 
and  a  generally  linear  pattern  for  Order  predicting  FSG. 

Openness  has  four  facets,  of  which  only  Creative  (01)  significantly  predicted  FSG  (r=.16,  p<.01),  as 
reported  in  Table  9.  Of  note,  this  facet  has  a  significant  effect  size  while  the  Openness  domain  was  non¬ 
significant  (r=.05),  which  speaks  to  the  importance  of  developing  and  operationally  using  narrower  facet 
scales  as  opposed  to  broader  domain  scales  when  predicting  occupational-specific  criteria.  The 
remaining  facets  of  Reflective  (02),  Scientific  Interest  (03),  and  Cultured  (04)  were  non-significant 
predictors  but  the  latter  two  had  small  but  noticeable  effects  sizes  that  were  close  in  magnitude  to  some 
of  the  significant  facet  predictors  (rs=.10).  An  examination  of  Figure  9  indicates  positive  liner 
relationships  with  FSG  for  both  Creative  and  Scientific  Interest,  however,  as  in  the  case  with  Sociability, 
the  facets  of  Reflective  and  Cultured  show  possible  nonlinear  relationships  with  the  criterion.  The 
former  showing  relatively  higher  scores  for  the  top  and  bottom  quintiles  and  the  latter  a  possible 
multiple  bend  curve.  Modeling  Cultured  in  a  polynomial  (or  quadratic)  regression  did  not  result  in  a 
significant  model;  however,  modeling  Reflective  as  a  quadratic  resulted  in  a  significant  predictor  model 
with  all  terms  significant  (p<. 05)  and  an  effect  size  of  R  =  .14,  an  improvement  over  the  r  =  -.06  linear 
correlation. 

The  last  domain  of  Machiavellianism  constitutes  the  three  facets  of  Envious  (Ml),  Individualistic  (M2), 
and  Self-Serving  (M3).  These  scales  are  shorter  than  most  of  the  other  facet  scales  (3  to  4  items  each) 
and  have  marginal  reliabilities  (as  range  from  .56  to  .77).  As  shown  in  Table  10,  none  of  these  facets 
were  significant  predictors  of  FSG.  Figure  10  displays  the  FSG  quintile  results  for  the  facets  and  reveal  no 
obvious  non-linear  (or  linear)  pattern  of  relationship. 

Regression  Modeling 

All  the  cognitive  (composites,  subtests)  and  many  of  the  non-cognitive  (domains,  facets)  scales  of  the 
AFOQT  Form  S  appear  to  be  quite  predictive  of  17D  UCT  Phase  II  FSG.  Additionally,  the  significant  non- 
cognitive  predictors  likely  predict  variance  of  FSG  that  is  largely  unique  to  that  predicted  by  the 
cognitive  measures.  Simply  choosing  one  predictor  would  be  effective,  but  would  not  allow  for  the 
unique  contribution  of  other  predictors.  Many  of  these  predictors  likely  covary  and  including  all  of  them 
would  be  redundant  and  unnecessary.  However,  some  may  uniquely  contribute  to  the  overall  possible 
predicted  variance.  Thus,  an  optimal  combination  of  some  number  of  significant  unique  predictors  is 
desired  for  maximizing  predictive  ability. 

A  series  of  multiple  regression  analyses  began  with  entering  all  predictors  with  significant-zero  order 
correlations  at  the  individual  scale  level  (see  Tabachnick  &  Fidell,  2001,  p.133,  for  an  explanation  on 
best-practice  procedures  for  the  model  building  process  using  multiple  regression,  i.e.,  variable 
inclusion/exclusion).  While  some  facet  scores  showed  promise  with  nonlinear  scaling  [i.e.,  Sociability 
(IE2)  &  Reflective  (02)],  model  building  began  with  linear  predictors  that  showed  significant  zero-order 
correlations.  Individual  test  scores  (subtests  &  facets)  were  exclusively  used  to  avoid  overlap  of  item 
content  among  variables  from  different  levels  (e.g.,  same  items  appearing  in  composite  and  subtest). 
This  resulted  in  a  19  variable  model  predicting  25%  of  the  FSG  variance.  This  model  is  useful  but  overly 
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complex  and  quite  redundant  as  evidenced  when  most  of  the  predictors  non-significantly,  uniquely 
contribute  to  prediction  of  FSG.  In  fact  only  MK,  GS,  and  Cl  were  significant  (p<. 05)  in  this  linear 
combination.  However,  predictors  that  are  non-significant  should  not  necessarily  be  excluded  because 
they  may  be  significant  in  other  combinations.  Therefore,  any  predictor  that  was  significant  at  p<. 20  was 
then  included  in  the  next  multiple  regression. 

The  next  model  included  seven  predictor  scores  (MK,  GS,  Cl,  WK,  1C,  IE3,  and  01)  and  predicted  23%  of 
the  FSG  variance.  This  resulted  in  only  a  slight  reduction  in  prediction  while  drastically  reducing  the 
complexity  of  the  model.  As  was  the  case  in  the  first  model  examined,  some  of  the  predictors  covary 
with  one  another  as  evidenced  by  three  of  the  seven  being  non-significant  (MK,  IE3,  &  01).  Retaining  the 
significant  predictors  (p<. 05)  for  the  next  multiple  regression,  the  third  model  included  WK,  1C,  GS,  and 
Cl.  This  model  predicted  almost  as  well,  22%  of  the  variance  with  all  four  predictors  significant  (p<. 05). 
Lastly,  those  facet  scales  that  showed  significant  quadratic  terms  predicting  FSG  (02  &  IE2)  were 
included  in  this  model.  02  and  its  squared  term  were  not  significant  predictors  in  the  model,  however, 
both  IE2  and  its  squared  term  were. 

The  model  shown  below  is  summarized  in  Table  11  and,  prior  to  cross-validation  on  an  independent 
sample,  was  the  recommended  model  for  17D  UCT  academic  performance  and  formulates  the  Cyber 
Composite  Score.  This  model  had  a  multiple  correlation  of  R  =  .48  and  predicted  23%  of  the  variance 
with  all  predictors  statistically  significant  (p<. 05).  The  parameter  estimates  ( b )  and  standardized  beta 
weights  ((3)  are  reported  in  Table  11. 


Validated  Initial  Cyber  Composite  Score  1  =  b0  +  biWK  +  b2GS  +  b3 1C  +  b4Cl  -  b5 1 E2  +  b6IE22  +  £ 


Cross-validation  (using  weights  from  development  (random  half)  sample ,  Bi-fold  design. 

Once  subsequent  data  became  available  for  cross-validating  the  Cyber  Composite  Score  (N=291),  the 
validated  cyber  composite  score  was  cross-validated  to  assess  prediction  independent  of  data  used  in 
initial  model  building  (sans  Type  II  error  due  to  chance  statistical  relationships).  The  IE2  terms  (linear  and 
quadratic  terms  for  Sociability)  were  not  significant  contributors  upon  cross  validation  (p>.05). 

Cross  validation  of  initial  cyber  composite  model: 

C17DXQ  =  70. 77+. 24*WK+. 37*GS+. 17 * IC+. 0013*C1 -. 0015* IE2+. 000000194* IE2sq; 

R  =  .396  (n  =  291  CV  sample). 

Once  the  initial  cyber  composite  score  was  cross-validated  to  produce  the  reduced  model  without  the  IE 
terms,  all  data  were  combined  (initial  validation  and  subsequent  data;  N=586)  and  then  randomly 
assigned  to  two  groups,  a  subsequent  validation  group  (to  replicate  the  model  building  process  of  the 
first  cross-validation  process)  and  a  cross-validation  sample  to  further  assess  prediction  independent  of 
data  used  in  model  building  in  a  bi-fold  design.  This  design  validates  the  reduced  model  with  one  sample 
and  cross-validates  the  model  with  the  other,  then  reverses  the  process.  So  both  samples  are  used  to 


1  Note.  WK  =  Word  Knowledge,  GS  =  General  Science,  1C  =  Instrument  Comprehension,  Cl  =  Achievement-Striving, 

IE2  =  Sociability,  £  =  normally  distributed  random  error  term. 
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validate  and  cross-validate  the  model  separately  for  replicability  of  results.  Below  are  the  regression 
models  for  the  bi-fold  cross-validation  process. 

Bi-fold  cross-validation: 

Bi-fold  1: 

C17DXL2  =  71+.20*WK+.26*GS+.17*IC+.0013*C1 

R  =  .405  (n  =  283  DEV  sample  2,  MYLIB.OQT17DB2).  All  linear  terms  significant  (p<.05). 

R  =  .431  (n  =  303  CV  sample  1,  MYLIB.OQT17DB). 

+AR  =  .026 

Bi-fold  2: 

C17DXL2b  =  68.25+. 30*WK+ .41*GS+.09*IC+.0013*C1 

R  =  .444  (n  =  303  DEV  sample  1,  MYLIB.OQT17DB).  All  linear  terms  significant  (p<.05)  except  1C. 

R  =  .396  (n  =  283  CV  sample  2,  MYLIB.OQT17DB2). 

-AR  =  .048 


Average  shrinkage,  -AR  =  .022 


Final  model  weights  based  on  full  sample 

The  final  cross-validated  model  shown  below  is  summarized  in  Table  12  and  is  the  recommended  model 
for  17D  UCT  academic  performance  and  formulates  the  Cyber  Composite  Score.  This  model  had  an 
average  observed  multiple  correlation  of  R  =  .423  average  cross-validity  of  R  =  4.01  before  any 
correction  for  range  restriction. 


Final  [Cross-Validated]  Cyber  Composite  Score  =  b0  +  biWK  +  bi GS  +  b3IC  +  b4Cl  +  £ 


Recommended  linear  model  with  full  sample  weights  applied: 

Final  CSS  =  C17DXL  =  69 . 6+  25*WK+ . 33*GS+  13*IC+ . 0013*C1 


Cut-Score  Recommendations 

In  an  attempt  to  discern  an  optimal  cut-score  for  the  Cyber  Composite  Score  (CCS),  the  expected  values 
of  FSG  (which  is  the  CCS)  are  plotted  against  the  actual  FSG  values  (Figure  11).  An  FSG  of  70  (on  the  y- 
axis)  is  considered  passing.  Note  only  those  who  made  it  all  the  way  through  the  course  are  represented 
in  this  sample  (although  not  all  passed  with  70  or  above).  Corrections  for  range  restriction  are  necessary 
for  accurate  estimates  of  cross-validity  due  attenuation  in  observed  validities. 

Plotting  the  CCS  against  FSG  enables  a  visual  examination  of  the  effect  of  imposing  various  cut-scores  in 
terms  of  prediction  model  hits  and  misses.  Several  likely  cut-scores  were  examined:  CCS  =  80,  81,  82,  83, 
84,  85,  and  86.  Hits  and  misses  are  evaluated  in  terms  of  Correct  Accepts  (a  power),  Correct  Rejects  ((3 
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power),  Incorrect  Accepts  (Type  I  error),  and  Incorrect  Rejects  (Type  II  error).  Chi  Square  analyses  (and 
Fisher's  Exact  tests)  were  conducted  for  the  seven  cut-scores  that  were  determined  from  the  visual 
examination  of  the  scatter  plot  (see  Table  13). 

Table  13  summarizes  the  results  of  the  Chi  Square  tests  for  the  various  cut-scores  of  80  through  86.  The 
Chi  Square  value  (x2),  the  Phi  Coefficient  (<$>),  Total  Power  (a  +  (3),  and  Total  Error  (Type  I  +  Type  II)  are 
reported  as  indicators  in  side-by-side  comparisons  of  the  cut  scores.  As  shown,  a  cut-score  of  80  is 
preferred  overall  and  for  each  indicator.  The  Chi  Square  statistic,  the  Phi  Coefficient,  and  Total  Power 
are  largest  and  Total  Error  is  smallest  for  this  cut-score.  Thus,  a  cut-score  of  CCS=80  is  recommended  for 
most  efficiently  increasing  accession  pool.  An  alternative  cut-score  of  83  is  recommended  if  reducing 
school  attrition  is  the  primary  goal.  Tables  14a  and  14b  reports  the  details  of  Chi  Square  analysis  for 
these  two  cut-scores. 

Subgroup  Mean  Score  Differences 

In  the  interest  of  diversity  and  fairness,  average  levels  of  CCS  were  computed  in  a  random  sample  of 
AFOQT  scores  and  compared  across  EEOC  protected  subgroups  that  appeared  in  sufficient  numbers  for 
statistical  analysis  (i.e.,  n  >  90).  The  comparisons  of  interest  are  sex,  ethnicity  (Hispanic),  and  Race.  Of 
the  groups  with  sufficient  sample  size,  sex,  ethnicity  (non-Hispanic  to  Hispanic),  race  (White-Black  & 
White-Asian)  were  compared.  Specifically  the  standardized  mean  CCS  difference  was  examined  for 
effect  size  (Cohen's  d )  and  statistical  significance  (95%  Confidence  Interval  not  containing  a  value  of  0). 
This  type  of  analysis  indicates  if  there  is  a  significant  difference  (p<. 05)  in  the  average  predictor  score 
between  two  groups  of  comparison  and  the  magnitude  of  that  differences,  expressed  in  standard 
deviation  units  (d). 

Table  15  reports  the  results  of  this  set  of  analyses.  All  of  the  four  comparisons  show  a  statistically 
significantly  higher  mean  score  for  the  majority  group  of  comparison  and  each  varied  as  to  the  effect 
size.  Male  to  Female  comparison  showed  a  significant  difference  with  an  effect  size  of  d=. 67  favoring 
males,  which  according  to  Cohen's  (1977)  arbitrary  heuristic,  is  considered  a  moderate  difference.  This  is 
not  unlike  Male-Female  differences  in  cognitive  abilities  tests  of  this  nature  and  is  similar  to  other 
studies  involving  the  AFOQT  Form  S  ( e.g .,  Hardison,  Sims,  &  Wong,  2010).  Non-Hispanic  to  Hispanic 
comparison  resulted  in  a  d-value  of  .17,  which  was  significant  but  the  lower  bound  of  the  95% 
Confidence  Interval  was  nearly  zero  (.01).  Note  that  the  minority  group  in  this  comparison  (Hispanic) 
includes  Whites  and  males,  whereas  the  majority  group  in  the  Male  to  Female  comparison  included 
Hispanics  and  non-Whites.  Only  two  minority  groups  where  large  enough  for  statistical  analysis  of  racial 
comparisons.  Here  Whites  comprise  the  majority  group.  The  White-Black  comparison  showed  a 
significant  d-value  of  1.04  favoring  Whites,  which  is  considered  a  large  effect  by  Cohen's  standard. 

Finally  the  White-Asian  comparison  showed  a  d-value  of  .34,  which  Cohen  considers  a  small  sized  effect. 
These  racial  differences  are  also  typical  in  effect  size  to  other  studies  of  the  AFOQT  Form  S  (e.g., 

Hardison  et  al.,  2010). 


DISCUSSION 


Summary  of  Findings 

Cognitive  and  non-cognitive  variables  of  the  AFOQT-S  were  examined  in  their  potential  to  predict  school 
outcomes  for  17D  UCT  Cyber  Defense  and  Cyber  Control  Officer.  All  cognitive  subtests  and  eight  of  20 
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non-cognitive  facet  variables  showed  statistically  significant  zero-order  correlations  with  school 
outcomes.  A  series  of  regression  analyses  with  decision  rules  in  line  with  best  practices  for  selection  and 
classification  guided  the  model  building  process  (Tabachnick  &  Fidel  I,  2001),  which  resulted  in  a 
recommended  17D  Composite  for  predicting  school  performance.  Cut-score  analyses  determined  two 
optimal  cut-scores  for  maximizing  on  classification  efficiency,  which  depend  on  current  needs  of  the  Air 
Force. 

Unique  Contributions  to  the  Predictor  Space 

The  recommended  model  includes  constructs  that  do  not  theoretically  or  empirically  overlap  in  the 
predictor  space  of  the  17D  School  outcome.  Included  in  the  model  are  linear  predictors,  i.e.,  WK  (verbal, 
cognitive  domain),  GS  (science,  cognitive  domain),  1C  (perceptual,  cognitive  domain),  and  Cl 
(achievement  motivation,  non-cognitive  domain).  This  relatively  simple  model  predicts  nearly  as  well  as 
the  more  complex  configurations  explored  in  the  model  building  process.  This  cross-validated  model 
should  be  more  stable  as  it  capitalizes  less  on  chance  relationships  of  the  predictor  variables  with  the 
criterion. 

It  is  also  interesting  to  note  that  math,  science,  and  technology  majors  and  curricula  are  currently 
targeted  as  selection  requirements  for  cyber  training,  however  tests  of  verbal  ability  (WK  &  VA)  were 
also  strong  predictors  of  school  performance.  Table  3  shows  WK  and  VA  (along  with  GS)  to  have  the 
three  highest  zero-order  correlations  with  FSG  among  the  cognitive  subtests  of  the  AFOQT.  Although 
this  may  in  part  be  due  to  indirect  range  restriction  on  quantitatively-oriented  coursework  as  the 
current  selection  method  (which  can  artificially  attenuate  observed  validities  of  math  and  science 
predictors),  it  speaks  to  the  importance  of  verbal  ability  in  cyber  training  success.  This  finding  has 
potentially  important  implication  for  classification  as  applicants  with  math  degrees  (which  presumably 
are  high  in  demand)  may  better  serve  the  Air  Force  in  other  AFSs  while  cyber  training  may  be  better 
staffed  by  those  high  in  verbal  ability. 

Summary  of  Recommendations 

Currently  there  is  a  classification  system  that  evaluates  if  the  candidate  has  the  required  coursework  or 
has  a  qualified  undergraduate  major.  This  system  seems  to  be  working  well  as  overall  attrition  for 
Phases  I  and  II  combined  is  around  6%;  however,  it  is  only  an  indicator  of  classes  taken,  not  the  quality 
of  performance  in  those  classes. 

The  CCS  is  recommended  to  be  used  in  two  possible  ways:  First,  as  a  counseling  tool  to  advise  those  as 
to  their  probability  of  success  in  Undergraduate  Cyberspace  Training.  This  could  be  conducted  at  the 
accession  source  at  the  time  of  job  selection,  typically  Junior  year  of  college  (at  least  for  AFROTC  and 
USAFA). 

Second,  as  a  tool  for  the  Officer  Accessions  Branch  (DPSIP  )  to  use  in  addition  to  the  existing 
classification  system.  With  the  CCS,  the  classification  officer  can  stack  rank  among  qualified  candidates 
those  with  the  highest  probability  of  success  in  training.  This  is  quite  appropriate  because  the  model  was 
developed  from  those  who  have  met  the  classification  qualification.  In  other  words,  the  model  was 
developed  from  a  sample  of  airmen  who  already  had  the  minimum  required  coursework  on  their  college 
transcripts.  As  the  model  becomes  more  fully  validated  on  subsequent  samples,  it  could  eventually 
replace  the  current  classification  system. 
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Limitations  and  Conclusion 


The  observed  validities  of  the  predictors  in  the  CCS  and  correlation  matrix  have  not  been  adjusted  for 
direct  range  restriction  of  the  sample  that  was  selected  for  officership  into  the  U.S.  Air  Force  or  indirect 
restriction  in  range  resulting  from  the  current  classification  system  that  selected  the  officers  into 
training  for  Air  Force  Officer  Cyberspace  Operations  /  Cyberspace  Control  School.  As  such,  the  observed 
correlations  are  artificially  suppressed  and  are  likely  larger  effects  than  shown. 
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Table  1. 

17D  AFS-Awarding  Final  School  Grade  Quintiles 


Total 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

N 

295 

53 

72 

75 

54 

41 

Range 

1  to  99 

51-81 

82-85 

86-88 

89-91 

92-98 

M 

85.9 

76.7 

83.7 

87.1 

90 

94 

SD 

6.4 

7.8 

1.1 

0.8 

0.9 

1.5 

Figure  1. 

Distribution  of  17D  AFS-Awarding  Final  School  Grade  Quintiles  (N=295). 
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Table  2. 

AFOQT  Composites  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Composite 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

PILOT 

62.0 

24.7 

.36*** 

47.5 

60.5 

63.1 

63.9 

78.6 

NAVTECH 

65.1 

25.0 

.35*** 

52.3 

62.4 

64.8 

69.1 

81.7 

ACAD 

65.0 

25.0 

.38*** 

50.8 

61.5 

66.0 

69.1 

82.5 

VERB 

60.3 

26.1 

.36*** 

45.6 

56.7 

61.1 

65.8 

76.4 

QUANT 

65.7 

24.9 

29*** 

55.2 

63.3 

65.5 

67.7 

81.0 

Notes.  ***  p  <  .001,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Scores  are  percentiles. 


Figure  2. 

AFOQT  Composites  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Table  3. 

AFOQT  Sub-Test  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Sub-Test 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Verbal  Analogies  (VA) 

17.6 

3.7 

.33*** 

15.7 

17.2 

17.5 

18.8 

19.6 

Arithmetic  Reasoning  (AR) 

17.6 

5.0 

.28*** 

15.6 

17.0 

17.8 

18.1 

20.5 

Word  Knowledge  (WK) 

17.5 

5.0 

.34*** 

14.9 

16.7 

17.9 

18.0 

20.6 

Math  Knowledge  (MK) 

18.0 

4.7 

25*** 

16.5 

17.5 

17.7 

18.2 

20.7 

Instrument  Comprehension  (1C) 

13.5 

4.8 

25*** 

11.3 

13.5 

14.2 

13.0 

15.7 

Block  Counting  (BC) 

12.6 

4.1 

.19** 

11.3 

12.6 

12.7 

12.3 

14.5 

Table  Reading  (TR) 

28.0 

7.1 

.19** 

25.9 

27.1 

27.8 

29.0 

31.1 

Aviation  Information  (Al) 

10.3 

3.6 

.28*** 

8.4 

10.3 

10.3 

10.7 

12.1 

General  Science  (GS) 

14.3 

3.3 

.34*** 

12.8 

13.9 

13.8 

15.5 

15.9 

Rotated  Blocks  (RB) 

10.2 

3.1 

.21*** 

9.3 

10.5 

10.3 

10.0 

11.4 

Hidden  Figures  (HF) 

11.3 

3.2 

.21*** 

10.5 

11.1 

11.4 

11.5 

12.3 

Notes.  ***  p  <  .001,  **  p  <  .01,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Scores  are  number 
correct  (totals  vary  by  subtest). 


Figure  3. 

AFOQT  Sub-Test  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Note.  Number  correct  on  the  y-axis  (totals  vary  by  subtest). 
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Table  4. 

SDI+  Domain  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Domain 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Agreeableness  (A) 

4700 

990 

.12* 

4588 

4653 

4738 

4713 

4842 

Neuroticism  (N) 

5278 

1084 

 2i*** 

5475 

5403 

5275 

5202 

4913 

Intro-Extraversion  (IE) 

5369 

1093 

-.06  ns 

5345 

5353 

5442 

5471 

5158 

Conscientiousness  (C) 

4658 

1053 

.19*** 

4485 

4559 

4710 

4721 

4876 

Openness  (O) 

5021 

977 

.05  ns 

5083 

4920 

4978 

4944 

5300 

Machiavellianism  (M) 

5143 

958 

-,01ns 

5103 

5250 

5090 

5154 

5087 

Notes.  ***  p  <  .001,  *  p  <  .05,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores  M=5000 
SD=1000. 


Figure  4. 

SDI+  Domain  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 
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Table  5. 

SDI+  Agreeableness  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Agreeableness 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Team  player  (Al) 

4728 

978 

.19*** 

4598 

4631 

4665 

4820 

5059 

Pleasant  (A2) 

4715 

1039 

.11  ns 

4549 

4713 

4766 

4681 

4866 

Considerate  (A3) 

4732 

1054 

.08  ns 

4680 

4650 

4820 

4767 

4733 

Helpful-Altruistic  (A4) 

4732 

997 

.07  ns 

4707 

4708 

4819 

4642 

4768 

Hyper-Competitive  (A5) 

5125 

1068 

-.12* 

5117 

5267 

5108 

5094 

4960 

Notes.  ***  p  <  .001,  *  p  <  .05,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores  M=5000 
SD=1000. 


Figure  5. 

SDI+  Agreeableness  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 
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Table  6. 

SDI+  Neuroticism  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Neuroticism 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Stress-Under-Pressure  (Nl) 

5284 

1083 

-  22*** 

5480 

5390 

5304 

5200 

4922 

Temperamental  (N2) 

5282 

1062 

-.17** 

5426 

5407 

5239 

5309 

4917 

Worry  (N3) 

5140 

1067 

-.13* 

5254 

5247 

5136 

5009 

4982 

Notes.  ***  p  <  .001,  **  p  <  .01,  *  p  <  .05,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores 
M=5000  SD=1000. 


Figure  6. 

SDI+  Neuroticism  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


7000 
6500 
6000 
5500 
5000 
4500 
4000 
3500 
3000 

N1  N2  N3 


H  quintilel 

■  q  u  i  nti  I  e2 

■  q  u  i  nti  I  e3 

■  q  u  i  nti  I  e4 

■  q  u  i  nti  I  e5 


Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 
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Table  7. 

SDI+  Intro-Extraversion  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Intro-Extraversion 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Unassertive  (1  El) 

5366 

1112 

-.09  ns 

5392 

5353 

5448 

5448 

5098 

Sociable  (IE2) 

4682 

1056 

-.03  ns 

4833 

4730 

4615 

4489 

4781 

Dominance  (IE3) 

4863 

954 

.13* 

4783 

4744 

4894 

4982 

4961 

Notes.  *  p  <  .05,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores  M=5000  SD=1000. 


Figure  7. 

SDI+  Intro-Extraversion  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 
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Table  8. 

SDI+  Conscientiousness  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Conscientiousness 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Achievement-Striving  (Cl) 

4664 

1066 

.21*** 

4446 

4586 

4704 

4739 

4914 

Order  (C2) 

4718 

1033 

.10  ns 

4670 

4601 

4790 

4734 

4833 

Notes.  ***  p  <  .001,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores  M=5000  SD=1000. 


Figure  8. 

SDI+  Conscientiousness  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


7000 

6500 

6000 

5500 

5000 

4500 

4000 

3500 

3000 


Cl 


C2 


■  quintilel 

■  q  u  i  nti  I  e2 

■  q  u  i  nti  I  e3 

■  q  u  i  nti  I  e4 

■  q  u  i  nti  I  e5 


Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 
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Table  9. 

SDI+  Openness  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Openness 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Creative  (01) 

4884 

989 

.16** 

4778 

4789 

4898 

4815 

5249 

Reflective  (02) 

5034 

1018 

-.06  ns 

5271 

4984 

4998 

4876 

5090 

Scientific  Interest  (03) 

5096 

1001 

.10  ns 

4911 

5084 

5060 

5142 

5363 

Cultured  (04) 

4937 

976 

-.10  ns 

5145 

4834 

4930 

4969 

4816 

Notes.  **  p  <  .01,  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores  M=5000  SD=1000. 


Figure  9. 

SDI+  Openness  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 


22 


Table  10. 

SDI+  Mach  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 


Machiavellianism 

M 

SD 

rXy 

quintilel 

quintile2 

quintile3 

quintile4 

quintile5 

Envious  (Ml) 

5227 

982 

-.06  ns 

5277 

5335 

5128 

5261 

5107 

Individualistic  (M2) 

5132 

973 

.04  ns 

4991 

5230 

5063 

5164 

5224 

Self-Serving  (MS) 

4996 

939 

.00  ns 

5005 

5068 

5006 

4916 

4943 

Notes.  rxy  is  the  uncorrected  correlation  with  final  school  grade.  Standard  scores  M=5000  SD=1000. 


Figure  10. 

SDI+  Mach  Facet  Mean  Scores  by  17D  Final  School  Grade  Quintile 
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Note.  Standard  scores  M=5000  SD=1000  on  the  y-axis. 
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Table  11. 

Initial  Cyber  Composite  Score. 


Parameter/Model 

b 

6 

t/F 

P 

R/R2 

Intercept 

70.77 

0 

24.58 

<.0001 

Word  Knowledge  (WK) 

0.24 

.19 

3.06 

.0025 

General  Science  (GS) 

0.37 

.19 

2.96 

.0025 

Instrument  Comprehension  (1C) 

0.17 

.13 

2.31 

.0033 

Achievement  Striving  (Cl) 

0.0013 

.21 

3.85 

.0218 

Sociable  (IE2) 

-0.0015 

-.25 

-2.12 

.0001 

Sociable2  (IE22) 

1.94'7 

.26 

2.17 

.0350 

Full  Model 

14.24 

<.0001 

.48/. 23 

Notes.  N=295.  b  =  unstandardized  parameter  estimate,  6  =  standardized  beta  weight,  t  =  parameter  statistic, 
F  =  model  statistic,  R  =  multiple  correlation,  R2  =  coefficient  of  determination. 


Table  12. 

Final  Cyber  Composite  Score  (CCS)  N=.  CSS  =  WK  +  GS  +  1C  +  Cl 


Parameter/Model 

b 

6 

t/F 

P 

R 

Intercept 

69.63 

0 

44.24 

<.001 

Word  Knowledge  (WK) 

0.25 

0.19 

4.17 

<.001 

General  Science  (GS) 

0.33 

0.18 

3.84 

<.001 

Instrument  Comprehension  (1C) 

0.13 

0.10 

2.40 

<.05 

Achievement  Striving  (Cl) 

0.0013 

0.21 

5.52 

<.001 

Full  Model 

31.6 

<.001 

.423 

Notes.  N=586.  b  =  unstandardized  parameter  estimate,  6  =  standardized  beta  weight,  t  =  parameter  statistic, 
F=  model  statistic,  R  =  multiple  correlation,  N=586. 


Table  13. 

Final  CCS  Model  Fit  Comparisons  for  Six  Cut-Scores 


CCS  Model 

FSGhat  =  WK  +  GS  +  1C  +  Cl 

Cut-score 

80* 

81 

82 

83** 

84 

85 

86 

X2 

114.00 

94.15 

37.53 

78.91 

42.80 

24.80 

25.19 

O 

.44 

.40 

.25 

.37 

.27 

.21 

.21 

Total  Power 

98.63 

98.46 

97.10 

94.53 

90.10 

84.30 

74.92 

Total  Error 

1.36 

1.53 

2.90 

5.46 

9.90 

15.70 

25.09 

Notes.  N=586,  *Best  fitting  CSS  cut-score  for  increasing  accession  pool.  **Best  fitting  CSS  cut-score  for  reducing 
attrition. 
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Table  14a. 

Hit  &  Miss  Rates  for  Final  CCS  Model  Predicting  FSG  -  Cut-Score  80 


At  or 

Above 

Base  rate 

70 

Incorrect  Reject 
(Type  II  Error) 
n=2 

0.34% 

Correct  Accept 
(a  power) 
n=575 

98.12% 

Actual 

Final 

School 

Below 

Correct  Reject 
(P  power) 

Incorrect  Accept 
(Type  1  Error) 

Grade 

Base  rate 

n=3 

n=6 

(FSG) 

70 

0.51% 

1.02% 

N=586 

Below  Cut  80 

At  or  Above  Cut  80 

Cyber  Composite  Score 
(Predicted  FSG) 


NOTE:  Only  data  from  those  with  a  FSG  shown  here  (made  it  through  school). 


Table  14b. 

Hit  &  Miss  Rates  for  Final  CCS  Model  Predicting  FSG  -  Cut-Score  83 


At  or 

Above 

Base  rate 

70 

Incorrect  Reject 
(Type  II  Error) 
n=30 

5.12% 

Correct  Accept 
(a  power) 
n=547 

93.34% 

Actual 

Final 

School 

Below 

Correct  Reject 
(P  power) 

Incorrect  Accept 
(Type  1  Error) 

Grade 

Base  rate 

n=7 

n=2 

(FSG) 

70 

1.19% 

0.34% 

N=586 

Below  Cut  83 

At  or  Above  Cut  83 

Cyber  Composite  Score 
(Predicted  FSG) 


NOTE:  Only  data  from  those  with  a  FSG  shown  here  (made  it  through  school). 
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Table  15. 

Standardized  Mean  Score  Differences  (Cohen's  d )  on  the  Predictor  Model  (Cyber  Composite  Score). 


Comparison 

Mmai 

SDmai 

Nmai 

Mmin 

SDmjn 

Nmin 

d 

95%  Cl  low 

95%  Clhigh 

Male-Female 

86.94 

2.55 

960 

85.24 

2.52 

180 

0.67* 

0.51 

NonHispanic-Hispanic 

86.69 

2.63 

1000 

86.25 

2.26 

97 

0.17* 

0.01 

White-Black 

87.12 

2.50 

805 

84.50 

2.74 

93 

1.04* 

0.87 

White-Asian 

87.12 

2.50 

805 

86.26 

2.45 

109 

0.34* 

0.17 

0.81 

Notes.  Ns=1097-898.  Subscript  maj  =  majority  group,  min  =  minority  group;  *p  <  .05. 
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Figure  11.  Plot  of  Final  School  Grade  (FSG)  by  Final  Cyber  Composite  Score  (CCS) 

Plot  of  FSG*CSS .  Legend:  A  =  1  obs,  B  =  2  obs,  etc. 
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Notes:  N=586.  Only  data  from  those  with  a  FSG  shown  here  (i.e.,  those  making  it  through  school) 
C17DXL  is  var  name  for  Final  CCS  in  the  AFPC/DSYX  database  (AF0QT  Historical  Masterfile), 
expressed  in  predicted  FSG  metric 
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